On the Error Probability of Model Selection for Classi cation

We consider model selection based on information criteria for classi cation. In classi cation, a class y is guessed from an attribute x based on the true conditional probability P (yjx), where x 2 X, y 2 Y , and X and Y are in nite and nite sets. In model selection, given examples, we select the model that minimizes an information criterion. The information criteria we address in this paper are expressed in the form of the empirical entropy plus a compensation term (k(g)=2)d(n), where k(g) is the number of independent parameters in a model g, d(n) is a function of n by which the information criterion is characterized, and n is the number of examples. We derive for arbitrary d( ) the asymptotically exact error probability in model selection. Although it was known for autoregressive processes that 2 d(n) = log log n is the minimum function of n such that the model selection satis es strong consistency, the problem whether the same thing holds for classi cation has been open. We solve this problem in the a rmative. Additionally, we derive for the d( ) that satisfy weak consistency the expected Kullback-leibler divergence between a true conditional probability P (yjx) and the conditional probability ^ P (yjx) estimated by the model selection and a parameter estimator. The derived value is k(g )=(2n), where g is a true model, and the accumulated value over n time instances is computed as (k(g )=2) log n+O(1), which implies the optimality of a predictive coding based on the model selection.